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WWC Study Reports are intended to support decision making; neither the What Works Clearinghouse 
(WWC) nor the U.S. Department of Education endorses any interventions. No single Study Report should 
be used as a basis for making policy decisions because (1) few studies are designed and implemented 
flawlessly and (2) all studies are tested on a limited number of participants, using a limited number of 
outcomes, at a limited number of times, so generalizing from one study to any context is very difficult. 

To highlight these issues, the WWC Study Reports describe in detail the specifics of each study, focusing 
primarily on studies that provide the best evidence of effects (randomized controlled trials). Systematic 
reviews of the evidence will be conducted to summarize the results of the individual studies. 

See the WWC Detailed Study Report (PDF) for more information about this study. 



Topic: Curriculum-Based Interventions for Increasing K-12 Math Achievement — 
Middle School 



Intervention: 
Research Design: 
Study Rating: 
Date Released: 



Saxon Math (Saxon Algebra 1/2 ) 
Quasi-Experimental Design with Stratified Matching 

o 
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Summary of Results: Crawford and Raia (1986) found that students in the intervention group 

scored significantly higher than students in the comparison group on math 
computation, but not on total math or math concepts. Since the analysis 
was done at the student level while the intervention assignment was at the 
classroom level, caution should be used in interpreting the results. 






= Meets Evidence Standards 



O 



= Meets Evidence Standards with Reservations 



Q 



= Does Not Meet Evidence Standards 



The What Works Clearinghouse (www.whatworks.ed.gov) was established in 2002 by the U.S. 
Department of Education’s Institute of Education Sciences to provide educators, policymakers, 
researchers, and the public with a central and trusted source of scientific evidence of what works in 
education. Please email all questions and comments to info@whatworks.ed.qov . The What Works 
Clearinghouse is administered by the U.S. Department of Education through a contract to a joint 
venture of the American Institutes for Research and the Campbell Collaboration . 
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What Is This Report About? 

This WWC report reviews a study of the effects 
of Saxon Algebra 1/2 on the mathematics 
achievement of 8th-grade students and a 
comparison of textbook content between the 
intervention and comparison curricula 
(Crawford & Raia, 1986). Crawford and Raia 
conducted three analyses with different subsets 
of the overall sample, but only one analysis is 
reviewed in this study report. In this matched 
sample, math achievement of 8th-grade students 
participating in the Saxon Algebra 1/2 
(intervention) classrooms was compared to math 
achievement of 8th-grade students participating 
in Scott-Foresman Mathematics (comparison) 
classrooms. 

This report summarizes the study and reviews 
its strengths and weaknesses. 

How Was the Study Conducted? 

Crawford and Raia conducted this study with 
four teachers from the Oklahoma City Public 
Schools (OCPS). All four teachers taught both 
the intervention and comparison curricula. 
Crawford and Raia selected a sample of students 
from the intervention and comparison groups 
that was matched exactly on pretest total math 
scores for their analyses. At the end of the 



intervention, Crawford and Raia administered 
the California Achievement Test (CAT) total 
math, math computation, and math concepts to 
all students. Crawford and Raia (1986) 
compared posttest scores between groups for the 
matched sample to determine if the intervention 
was effective. 

Students in the intervention group participated 
in Saxon Algebra 1/2. Students in the 
comparison group participated in Scott- 
Foresman Mathematics. The intervention was 
piloted during the 1984-1985 academic year. 

What Did the Study Find? 

The WWC only reviewed the evidence related 
to the total math, math computation, and math 
concepts posttest scores of the intervention and 
comparison groups. The analysis (see Figures 
la-c) showed that students in the intervention 
group scored significantly higher than students 
in the comparison group on math computation, 
but not on total math or math concepts. 

The sample size for this study’s matched sample 
was small, making inferences problematic. 

Since the analysis was done at the student level 
while the intervention assignment was done at 
the classroom level, caution should be used in 
interpreting the results. 
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Figure la. Impact Calculated by Crawford and Raia (1986): CAT Total 
Math Posttest Scores 
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Note. The analyses are done at the level of the student, which does not match the unit of 
assignment (class). Crawford and Raia (1986) reported that the intervention group posttest scores 
did not differ significantly from the comparison group posttest scores (p = ns). CAT = California 
Achievement Test. 
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Figure 1b. Impact Calculated by Crawford and Raia (1986): CAT Math 
Computation Posttest Scores 
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Note. The analyses are done at the level of the student, which does not match the unit of 
assignment (class). Crawford and Raia (1986) reported that the intervention group posttest scores 
were significantly higher than the comparison group posttest scores (p = .05). CAT = California 
Achievement Test. 
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Figure 1c. Impact Calculated by Crawford and Raia (1986): CAT Math 
Concepts Posttest Scores 
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Note. The analyses are done at the level of the student, which does not match the unit of 
assignment (class). Crawford and Raia (1986) reported that the intervention group scores did not 
differ significantly from the comparison group scores (p = ns). CAT = California Achievement Test. 



■ Cost information: not reported for this study, 
but extensive cost information available at 
www.saxonpublishers.com. 

■ Intervention Developer Contact: 

Saxon Publishers 
2600 John Saxon Blvd. 

Norman, OK 73071 
(800) 284-7019 
info@saxonpublishers.com 
www.saxonpublishers.com 



How Can You Find Out More? 

■ To learn more about this study, read the 
detailed report (PDF) . 

■ See reports on other studies of Saxon Math . 



Report Production 

Date created: March 4, 2005 



Topic area reviewed under: Curriculum-Based 
Interventions for Increasing K-12 Math 
Achievement — Middle School 
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WWC Study Ratings 3 : Crawford and Raia (1986) 



Causal Validity: Meets WWC Evidence Standards, a Quasi-Experimental Design with Stratified 

Matching 

Students in the intervention group ( Saxon Algebra 1/2) were compared to students in the comparison group (Scott-Foresman 
Mathematics). It is not clear how teachers and students were identified for the study or how they were placed in groups. This 
study report focuses on the matched sample within the third of three studies. Crawford and Raia (1986) matched the students 
on pretest California Achievement Test (CAT) total math scores and compared posttest total math, math computation, and 
math concepts scores between groups. It does not appear that any teachers or students left the study. Crawford and Raia do 
not report any extraneous events that might have confounded the intervention’s effects. 



Other Study 
Characteristics 


Study Rating 


Study-Specific Information 


Intervention Fidelity 


• 


Saxon Math meets the definition of Middle School Math. Crawford and Raia do not 
provide information about the implementation of the Saxon Math program. There 
does not appear to be a measure of teacher implementation, and teacher training was 
not discussed. 


Outcome Measures 


• • 


One achievement measure was reported in the study, the CAT, which is a nationally 
normed, standardized test, has acceptable reliability and validity, and appears to align 
to the intervention. 


People, Settings, and 
Timing 


• 


The study included a relevant middle school math sample but it was homogeneous 
with respect to students and settings. The study included only one type of 
achievement outcome measure, but it was implemented at a time appropriate to 
capture the effect of the intervention. 


Testing within 
Subgroups 


• 


The effect of Saxon Algebra 1/2 on students’ mathematics achievement was tested 
across the entire matched sample, but was not tested within subgroups of the sample 
or variations across settings. 


Analysis 


• 


The unit of assignment (class) was not the same as the unit of analysis (student). The 
sample sizes were small for the matched sample in this study, making inferences 
problematic. 


Statistical Reporting 


• • 


Crawford and Raia report unadjusted means and standard deviations at the student 
level. Effect sizes for the outcome measure could be calculated using a standard 
formula if means and standard deviations were reported at the level of the class for 
the matched sample in this study. 



Note. •• Fully meets criteria; • Meets minimum criteria; X Does not meet criteria. 

a For more information on the criteria used to rate this study, see the WWC Evidence Standards: Middle-School Math . 
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